Skip to content

Generalize focal PG core build to a parameterized PG13-17 pipeline - #1185

Open
kemalbuyukkaya wants to merge 15 commits into
developfrom
pg-focal
Open

Generalize focal PG core build to a parameterized PG13-17 pipeline#1185
kemalbuyukkaya wants to merge 15 commits into
developfrom
pg-focal

Conversation

@kemalbuyukkaya

@kemalbuyukkaya kemalbuyukkaya commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

What

A standalone, signed pipeline that rebuilds PostgreSQL core .deb packages for Ubuntu 20.04 (focal) for every focal-buildable major: PostgreSQL 13, 14, 15, 16, 17.

PGDG stopped shipping focal binaries once focal reached EOL standard support (the last official builds were e.g. 16.9-1.pgdg20.04+1). These five majors still receive newer upstream minors than their last focal build, so we rebuild them ourselves to give focal users current minors with a working, focal-native JIT.

The approach combines the newer upstream orig.tar.bz2 with the frozen focal-era debian/ packaging for each major (its default toolchain is focal's LLVM 10, so the resulting JIT depends on libllvm10/libssl1.1 rather than the clang-19/llvm-19 that newer packaging requires), then signs with the existing debsigner (debsigs --sign=maint).

This is intentionally separate from the Citus extension flow (citus_package / build-package.yml), which assumes PostgreSQL itself comes from PGDG.

Changes

scripts/build_pg_focal — single parameterized build entrypoint

  • PG_MAJOR selects the major (13–17).
  • PG_UPSTREAM_VERSION auto-resolves to the latest minor from the live PGDG pool (still pinnable) — no edit needed when a new minor is released; orig/debian checksums auto-resolve from the official .dsc unless pinned.
  • DEBIAN_BASE comes from a static, frozen per-major map (focal is EOL, so the last focal packaging never changes), overridable via PG_DEBIAN_BASE.
  • Patch drift is handled declaratively via PG_DROP_PATCHES (default hurd-iovec, a no-op where absent); the quilt-series gate stays as the fail-loud safety net so future drift fails loudly.
  • Collects debug-symbol packages: dpkg-buildpackage emits them as .ddeb on Ubuntu, which the old *.deb collection dropped. They are now shipped as .deb (Debian convention, identical format) so they flow through signing/verification/upload like every other package.

dockerfiles/focal-pg-builder/ — one generic builder image (focal-pgdg main 13 14 15 16 17), built once and reused for every major.

scripts/smoke_test_focal_debs — install smoke test for the built set

The build jobs only prove the packages exist, are signed and have unique names — none of that proves they are installable. This installs the whole shipped set (including -dbgsym) into a stock ubuntu:20.04, asserts every package landed at its exact built version (so apt used our binaries and did not silently substitute a same-named package), creates and starts a cluster for every installed major, and checks pg_jit_available() on each — a regression in the frozen-packaging / LLVM-10 strategy that silently disabled JIT would otherwise ship unnoticed.

It also encodes the one external requirement rather than leaving it as tribal knowledge: the server packages need postgresql-common (>= 252~), but focal only ships 214ubuntu0.1, so the PGDG archive's main component is enabled to pull 278.pgdg20.04+1. Only main — never the per-major components, which still carry PGDG's own libpq5/libecpg6 for focal and would shadow the packages under test.

.github/workflows/build-pg-focal.yml

  • A setup job computes the matrix: a single major via workflow_dispatch, or 13–16 by default. PG17 is fully supported but kept out of the default set (not needed up front) — dispatch pg_major=17 to build it.
  • build-and-sign matrixes over majors (fail-fast: false): build → sign (citusdata/packaging:debsigner) → verify _gpgmaint → upload a per-major artifact. Output is namespaced per major (packages/focal/pg<major>/); the sign step mounts that parent so the signer's /packages/*/*.deb glob matches.
  • A new assemble job produces the combined set to publish (see below).
  • A new install-smoke-test job runs scripts/smoke_test_focal_debs against that combined set in a clean focal container.
  • Runs on every push to pg-focal and on workflow_dispatch. Top-level permissions: contents: read — no job writes back to the repository (artifact up/download use the Actions runtime token, not GITHUB_TOKEN).

Co-installable combined set (postgresql-all-focal)

Each major's source builds six single-instance system libraries that share one package name but carry a per-major version — libpq5, libpq-dev, libpgtypes3, libecpg6, libecpg-dev, libecpg-compat3 (plus their -dbgsym). They cannot be co-installed, and the newest copy satisfies every major's >= dependency.

The assemble job therefore keeps all per-major packages from every major but the shared libraries from the highest major only, guards against duplicate package names, and uploads a flat, de-duplicated postgresql-all-focal artifact. Packages are copied byte-for-byte, preserving their signatures. Per-major artifacts are also retained for traceability.

Validation

Built end-to-end for all five majors (real compiles on focal) — including PG17, which is supported on demand even though it is not in the default matrix. Each produces focal-native packages (libllvm10/libssl1.1, working JIT); hurd-iovec is dropped only on 16/17:

Major Auto minor Frozen base JIT dep Patch dropped
13 13.23 13.21-1.pgdg20.04+1 libllvm10 (none)
14 14.23 14.18-1.pgdg20.04+1 libllvm10 (none)
15 15.18 15.13-1.pgdg20.04+1 libllvm10 (none)
16 16.14 16.9-1.pgdg20.04+1 libllvm10 hurd-iovec
17 17.10 17.5-1.pgdg20.04+1 libllvm10 hurd-iovec
  • dbgsym: ~11 debug-symbol packages collected per major.
  • Combined set (validated over all five majors, i.e. a superset of the default 13–16 run): 72 packages, no duplicate package names, exactly one 17.10 copy of each shared library, postgresql-13..17 all present. A default run ships the same shape for 13–16, with the shared libraries taken from PG16.
  • Co-install (install-smoke-test): PG 13–17 install together on a clean focal container with no conflicts; every package installs at its exact built version; server binaries report 13.23 / 14.23 / 15.18 / 16.14 / 17.10; a single libpq5 17.10; all 31 dbgsym packages also install; all five clusters start on distinct ports and report pg_jit_available() = t.

The signing step itself only runs in CI (needs the deployed citusdata/packaging:debsigner image + PACKAGING_SECRET_KEY/PACKAGING_PASSPHRASE).

Notes

  • 11/12 are intentionally excluded (EOL — no minor newer than their last focal build); 18 never shipped on focal.
  • No apt index is generated; the combined .deb set is published as-is.
  • Consumers need the PGDG archive's main component enabled (apt-archive.postgresql.org, suite focal-pgdg): the server packages require postgresql-common (>= 252~) and focal only ships 214ubuntu0.1. Enable main only — the per-major components would shadow these packages. This is exercised by scripts/smoke_test_focal_debs.

Supersedes #1182.

PGDG dropped PostgreSQL 16 binaries for Ubuntu 20.04 (focal) after focal
reached EOL standard support (last official focal build: 16.9-1.pgdg20.04+1).
This adds a standalone, signed pipeline that rebuilds newer 16.x core packages
for focal.

Approach (validated locally; produces the full 13-package set that installs and
runs on stock focal with working JIT):
- Combine the newer upstream orig.tar.bz2 with the focal-era debian/ packaging
  (16.9-1.pgdg20.04+1), whose default toolchain (clang/llvm-dev = LLVM 10)
  yields focal-native dependencies (libicu66, libssl1.1, libldap-2.4-2,
  libllvm10) instead of the clang-19/llvm-19 required by newer packaging.
- Restore the removed focal-pgdg build tooling (debhelper 13, dh-exec,
  postgresql-common-dev) from apt-archive.postgresql.org.
- Drop the obsolete hurd-iovec patch (merged upstream as of 16.14) and gate on
  the full quilt series applying cleanly so future drift fails loudly.
- Sign with the existing debsigner image (debsigs --sign=maint), using the
  pg-azure-storage signing secrets (PGAZ_PACKAGE_SECRET_KEY /
  PGAZ_PACKAGE_PASSPHRASE), matching build-pgazure-nightlies.yml.

The minor version is parameterized: set PG_UPSTREAM_VERSION (workflow input
pg_upstream_version) to build e.g. 16.15; the orig/debian checksums are
auto-resolved from the official .dsc unless pinned.

Files:
- dockerfiles/pg16-focal-builder/Dockerfile: focal builder image
- scripts/build_pg16_focal: fetch/verify/assemble/build entrypoint
- .github/workflows/build-pg16-focal.yml: build -> sign -> verify -> upload
The packaging signing secrets in this repo (incl. PGAZ_PACKAGE_SECRET_KEY) are
stored as raw ASCII-armored keys, not base64. import_and_sign assumed base64 and
ran `base64 -d` first, which fails on armored input ("base64: invalid input" ->
"no valid OpenPGP data found" -> "secret key not available").

Detect the format: import ASCII-armored keys directly, otherwise base64-decode
as before (backward compatible). Also verify a PRIVATE key was actually imported
and exit non-zero with an actionable message if only a public key is present, so
debsigs never silently emits unsigned packages.

Validated in the xenial debsigner image: armored private key -> signs
(_gpgmaint added); base64 key -> signs; armored public-only key -> exits 78.
build-package.yml and build-package-test.yml trigger on every branch
(branches: "**") and run the Citus extension build plus test_build_packages,
which is unrelated to the PostgreSQL-core focal pipeline and fails here on a
pre-existing PACKAGING_PASSPHRASE mismatch. Exclude pg16-focal via
branches-ignore so it no longer blocks this work; workflow_dispatch stays.
Roll back the earlier workarounds now that signing uses the prebuilt
citusdata/packaging:debsigner image:
- dockerfiles/debsigner/scripts/import_and_sign: back to upstream (we no longer
  build our own signer, so the armored-key handling is unnecessary).
- build-package.yml / build-package-test.yml: restore branches: "**" (drop the
  pg16-focal branches-ignore guard) to avoid touching shared extension CI.

These three files now match develop; only the PostgreSQL-core focal pipeline
remains in this branch.
The signer images are maintained out-of-band (not built by this repo's image
pipeline), so the deployed citusdata/packaging:debsigner has drifted from
dockerfiles/debsigner. Building our own signer from that source could not import
the same signing key that signs every other Citus package, while the deployed
image does (pg-azure-storage nightlies are green with PGAZ_PACKAGE_SECRET_KEY).

Use the prebuilt citusdata/packaging:debsigner with a Docker Hub login and pipe
the passphrase via stdin + env, mirroring citus_package.sign_packages. Pin the
job to ubuntu-20.04 to match the green pg-azure-storage signing pipeline.
Revert the ubuntu-20.04 pin to ubuntu-latest to stay future-proof as GitHub
retires the hosted ubuntu-20.04 image. Signing uses the prebuilt
citusdata/packaging:debsigner image (same as all-citus, which signs fine on
ubuntu-latest), so the runner version is not the relevant factor.
PGAZ_PACKAGE_SECRET_KEY is only referenced on the pg-azure-storage branches and
did not import in our run. Switch to the common signing secrets used across the
other pipelines (PACKAGING_SECRET_KEY / PACKAGING_PASSPHRASE) to match the
standard convention.
Replace the PG16-only focal pipeline with a single parameterized one keyed
on PG_MAJOR, covering every focal-buildable major (PG 13, 14, 15, 16, 17).
PGDG dropped focal binaries once focal reached EOL; these are the majors
that still receive new upstream minors beyond their last focal build.

- scripts/build_pg16_focal -> scripts/build_pg_focal: add PG_MAJOR; auto-
  resolve the latest minor per major from the live pool (PG_UPSTREAM_VERSION
  override stays); static frozen DEBIAN_BASE map per major (focal is EOL, so
  these never change) with a PG_DEBIAN_BASE override; replace the hardcoded
  hurd-iovec drop with a declarative PG_DROP_PATCHES list (default hurd-iovec,
  a no-op where absent) so future drift is a one-line, no-code change. The
  quilt-series gate stays as the fail-loud safety net.
- dockerfiles/pg16-focal-builder -> dockerfiles/focal-pg-builder: one generic
  builder image (focal-pgdg main 13 14 15 16 17); built once, reused per major.
- build-pg16-focal.yml -> build-pg-focal.yml: a setup job computes the matrix
  (single major via workflow_dispatch, or all 13-17), build-and-sign matrixes
  over majors with fail-fast: false; sign/verify/upload are unchanged but
  namespaced per major. Trigger on the pg-focal branch.

Validated end-to-end for all five majors (full 13-package set each, depending
on focal-native libllvm10/libssl1.1, hurd-iovec dropped only on 16/17):
  13.23, 14.23, 15.18, 16.14, 17.10.
The debsigner entrypoint signs exactly "/packages/*/*.deb" (one directory
level deep) and, with no nullglob, passes the unexpanded literal to debsigs
when nothing matches -> "File /packages/*/*.deb does not exist".

The build writes to packages/focal/pg<major>/, so mounting ${PWD}/packages
left the debs one level too deep (/packages/focal/pg<major>/*.deb) and the
glob matched nothing. Mount ${PWD}/packages/focal instead, so the signer
sees /packages/pg<major>/*.deb. Verified against citusdata/packaging:debsigner.
Two issues with the multi-major focal output:

1. Debug-symbol packages were missing. dpkg-buildpackage emits them as .ddeb
   on Ubuntu, but the collection step copied only *.deb, so every dbgsym was
   silently dropped. Collect them too, renamed .ddeb -> .deb (Debian's own
   convention; identical on-disk format), so they flow through the existing
   debsigs signing, _gpgmaint verification and artifact upload unchanged.

2. Installing PG 13-17 on one machine conflicted on the shared libraries.
   Each major's source builds six single-instance system libraries with the
   same package name but a per-major version -- libpq5, libpq-dev, libpgtypes3,
   libecpg6, libecpg-dev, libecpg-compat3 (plus their -dbgsym) -- so they
   cannot be co-installed. The newest copy satisfies every major's ">="
   dependency. Add an "assemble" job that, after the per-major matrix, keeps
   all per-major packages from every major but the shared libraries only from
   the highest major present, guards against duplicate package names, and
   uploads a flat, de-duplicated postgresql-all-focal artifact. Per-major
   artifacts are retained for traceability. Packages are copied byte-for-byte,
   preserving their signatures.

Validated end-to-end (PG 13-17): dbgsym now collected (~11 per major); the
combined set is 72 packages with no duplicate names and a single 17.10 copy of
each shared lib; PG 13-17 co-install on a clean focal container with no
conflicts and all server binaries reporting the expected versions.
The assemble step assembled correctly but failed under set -euo pipefail:
the diagnostic loop used `[[ ... ]] && echo ...`, which returns non-zero on
the last package when it isn't a shared lib; piped into sort that made the
whole step exit 1. Use an explicit `if` so the loop ends with status 0.

Also drop `cp -n` (newer coreutils warns it's non-portable, and the flag is
unnecessary -- after de-dup no destination filename collides) for plain `cp`.
PG17 is not needed up front, so the default "all" matrix now builds
13..16 only. PG17 stays fully supported and selectable -- dispatch with
pg_major=17 to build it; scripts/build_pg_focal and the builder image
still carry the 13..17 support they always had.

The assemble job derives the newest major from the artifacts it actually
receives, so the combined set now takes the six single-instance shared
libraries (libpq5, libpq-dev, libpgtypes3, libecpg6, libecpg-dev,
libecpg-compat3) from PG16 instead of PG17 with no code change.

Also align action versions with develop: actions/checkout v4 -> v6 and
docker/login-action v2 -> v4 (v2 runs on the deprecated Node 16 runtime).
Comment thread .github/workflows/build-pg-focal.yml Fixed
Comment thread .github/workflows/build-pg-focal.yml Fixed
Comment thread .github/workflows/build-pg-focal.yml Fixed
The pipeline verified that packages exist, are signed, and carry unique
names, but never that they can actually be installed. In particular the
combined set ships the six single-instance shared libraries from only the
highest major, assuming e.g. libpq5 16.x satisfies PG13's "libpq5
(>= 13~~)". That assumption was untested.

New install-smoke-test job installs the whole shipped set (including
-dbgsym) into a stock ubuntu:20.04, then asserts:

  * every shipped package installed at its exact version, proving apt
    used our binaries rather than substituting same-named ones,
  * postgresql-common resolved to 278.pgdg20.04+1,
  * a cluster per installed major starts and answers queries,
  * pg_jit_available() is true on every cluster,
  * clusters coexist on distinct ports.

The JIT assertion matters because the failure is silent: if WITH_LLVM
resolves empty in postgresql.mk, configure just omits --with-llvm and the
build still produces a complete, signed, installable set with no
llvmjit.so. Nothing else in the pipeline would notice.

This is not a self-containment test. Focal ships postgresql-common
214ubuntu0.1 while the server packages need >= 252~, so the PGDG archive's
"main" component is enabled -- consumers need it too, which is why it is
encoded here rather than left as tribal knowledge.

The logic lives in scripts/smoke_test_focal_debs, mirroring how
scripts/build_pg_focal is run in a container, so it can be executed
locally against a downloaded artifact.
Comment thread .github/workflows/build-pg-focal.yml Fixed
Comment thread .github/workflows/build-pg-focal.yml Fixed
@kemalbuyukkaya
kemalbuyukkaya marked this pull request as ready for review July 29, 2026 13:57
CodeQL flagged every job in the workflow for not limiting the
permissions of GITHUB_TOKEN (alerts 13-17, rule
actions/missing-workflow-permissions).

No job writes back to the repository: they check out sources, build and
sign inside containers, and exchange artifacts. actions/upload-artifact
and actions/download-artifact v4 use the Actions runtime token rather
than GITHUB_TOKEN, and every download here is same-run, so read-only
contents covers all four jobs. A single top-level block clears all five
alerts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 38a0aa7c-f6f0-4d5f-b1a9-568d4c7e6729
@kemalbuyukkaya

Copy link
Copy Markdown
Collaborator Author

@microsoft-github-policy-service agree company="Microsoft"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants